Skip to content

feat: Confirm the OVN maintenance-worker periodic (ovn_maintenance_periodics) fires in our deployment PUC-2030 - #2262

Merged
nidzrai merged 1 commit into
mainfrom
neutron_maintenance_worker_check
Sep 4, 2026
Merged

feat: Confirm the OVN maintenance-worker periodic (ovn_maintenance_periodics) fires in our deployment PUC-2030#2262
nidzrai merged 1 commit into
mainfrom
neutron_maintenance_worker_check

Conversation

@nidzrai

@nidzrai nidzrai commented Sep 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

#2293

Neutron chart upgrade introduced CrashLoopBackOff coz of missing config map . the pr fixes that

neutron-ovn-maintenance-worker-6f447d55c9-98w6d                   0/1     CrashLoopBackOff   5 (66s ago)   4m48s
neutron-periodic-worker-774cc5bd67-qbx4x                          0/1     CrashLoopBackOff   5 (60s ago)   4m48s
QoS Class:                   Burstable
Node-Selectors:              openstack-control-plane=enabled
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m8s                   default-scheduler  Successfully assigned openstack/neutron-ovn-maintenance-worker-6f447d55c9-98w6d to 1327172-hp1
  Normal   Pulling    5m7s                   kubelet            spec.initContainers{init}: Pulling image "quay.io/airshipit/kubernetes-entrypoint:latest-ubuntu_noble"
  Normal   Pulled     5m7s                   kubelet            spec.initContainers{init}: Successfully pulled image "quay.io/airshipit/kubernetes-entrypoint:latest-ubuntu_noble" in 93ms (93ms including waiting). Image size: 112391032 bytes.
  Normal   Created    5m7s                   kubelet            spec.initContainers{init}: Created container: init
  Normal   Started    5m7s                   kubelet            spec.initContainers{init}: Started container init
  Normal   Pulling    4m33s                  kubelet            spec.initContainers{ovn-neutron-init}: Pulling image "quay.io/airshipit/neutron:2026.1-ubuntu_noble"
  Normal   Started    4m26s                  kubelet            spec.initContainers{ovn-neutron-init}: Started container ovn-neutron-init
  Normal   Created    4m26s                  kubelet            spec.initContainers{ovn-neutron-init}: Created container: ovn-neutron-init
  Normal   Pulled     4m26s                  kubelet            spec.initContainers{ovn-neutron-init}: Successfully pulled image "quay.io/airshipit/neutron:2026.1-ubuntu_noble" in 93ms (7.122s including waiting). Image size: 157245781 bytes.
  Normal   Pulled     4m24s                  kubelet            spec.containers{neutron-ovn-maintenance-worker}: Successfully pulled image "quay.io/airshipit/neutron:2026.1-ubuntu_noble" in 112ms (252ms including waiting). Image size: 157245781 bytes.
  Normal   Pulled     4m23s                  kubelet            spec.containers{neutron-ovn-maintenance-worker}: Successfully pulled image "quay.io/airshipit/neutron:2026.1-ubuntu_noble" in 90ms (189ms including waiting). Image size: 157245781 bytes.
  Normal   Pulled     4m7s                   kubelet            spec.containers{neutron-ovn-maintenance-worker}: Successfully pulled image "quay.io/airshipit/neutron:2026.1-ubuntu_noble" in 728ms (728ms including waiting). Image size: 157245781 bytes.
  Normal   Pulled     3m38s                  kubelet            spec.containers{neutron-ovn-maintenance-worker}: Successfully pulled image "quay.io/airshipit/neutron:2026.1-ubuntu_noble" in 114ms (114ms including waiting). Image size: 157245781 bytes.
  Normal   Created    2m55s (x5 over 4m24s)  kubelet            spec.containers{neutron-ovn-maintenance-worker}: Created container: neutron-ovn-maintenance-worker
  Normal   Pulled     2m55s                  kubelet            spec.containers{neutron-ovn-maintenance-worker}: Successfully pulled image "quay.io/airshipit/neutron:2026.1-ubuntu_noble" in 106ms (106ms including waiting). Image size: 157245781 bytes.
  Normal   Pulling    86s (x6 over 4m24s)    kubelet            spec.containers{neutron-ovn-maintenance-worker}: Pulling image "quay.io/airshipit/neutron:2026.1-ubuntu_noble"
  Warning  Failed     86s (x6 over 4m24s)    kubelet            spec.containers{neutron-ovn-maintenance-worker}: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "/tmp/neutron-ovn-maintenance-worker.sh": is a directory
  Normal   Pulled     86s                    kubelet            spec.containers{neutron-ovn-maintenance-worker}: Successfully pulled image "quay.io/airshipit/neutron:2026.1-ubuntu_noble" in 168ms (168ms including waiting). Image size: 157245781 bytes.
  Warning  BackOff    18s (x19 over 4m22s)   kubelet            spec.containers{neutron-ovn-maintenance-worker}: Back-off restarting failed container neutron-ovn-maintenance-worker in pod neutron-ovn-maintenance-worker-6f447d55c9-98w6d_openstack(0c785f7d-9c74-410f-92ec-3b29f03757b1)
Flag --oidc-use-pkce has been deprecated, use --oidc-pkce-method instead. For the most providers, you don't need to set the flag.
Defaulted container "neutron-ovn-maintenance-worker" out of: neutron-ovn-maintenance-worker, init (init), ovn-neutron-init (init)

(openstack)

What does this change do?

we see neutron-ovn-maintenance-worker up and running , below are the logs

Upgrade impact

We see the netdev tick , which runs after every 60s

2026-09-04 09:33:58.533 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_ha_failover finished in 0.015 seconds
2026-09-04 09:33:58.535 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_lrouter_ports_ext_ids_name_prefix finished in 0.001 seconds
2026-09-04 09:33:58.535 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_lrouter_ports_ext_ids_name_prefix finished in 0.001 seconds
2026-09-04 09:33:58.538 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_mac_aging_settings finished in 0.002 seconds
2026-09-04 09:33:58.538 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_mac_aging_settings finished in 0.002 seconds
2026-09-04 09:33:58.545 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_router_distributed_flag finished in 0.006 seconds
2026-09-04 09:33:58.545 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_router_distributed_flag finished in 0.006 seconds
2026-09-04 09:33:58.546 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_router_static_routes finished in 0.001 seconds
2026-09-04 09:33:58.546 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_router_static_routes finished in 0.001 seconds
2026-09-04 09:33:58.558 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_security_group_with_address_group finished in 0.011 seconds
2026-09-04 09:33:58.558 10 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] OVN maintenance task update_security_group_with_address_group finished in 0.011 seconds
2026-09-04 09:34:49.495 10 WARNING neutron_understack.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] NETDEV reconcile tick (host=neutron-ovn-maintenance-worker-759f7d8784-7qqsx)
2026-09-04 09:34:49.495 10 WARNING neutron_understack.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] NETDEV reconcile tick (host=neutron-ovn-maintenance-worker-759f7d8784-7qqsx)
2026-09-04 09:35:49.496 10 WARNING neutron_understack.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] NETDEV reconcile tick (host=neutron-ovn-maintenance-worker-759f7d8784-7qqsx)
2026-09-04 09:35:49.496 10 WARNING neutron_understack.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] NETDEV reconcile tick (host=neutron-ovn-maintenance-worker-759f7d8784-7qqsx)
2026-09-04 09:36:49.496 10 WARNING neutron_understack.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] NETDEV reconcile tick (host=neutron-ovn-maintenance-worker-759f7d8784-7qqsx)
2026-09-04 09:36:49.496 10 WARNING neutron_understack.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] NETDEV reconcile tick (host=neutron-ovn-maintenance-worker-759f7d8784-7qqsx)
2026-09-04 09:37:49.496 10 WARNING neutron_understack.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] NETDEV reconcile tick (host=neutron-ovn-maintenance-worker-759f7d8784-7qqsx)
2026-09-04 09:37:49.496 10 WARNING neutron_understack.maintenance [None req-fd3ace59-0bf3-4675-9e69-9caf6cec79f7 - - - - - -] NETDEV reconcile tick (host=neutron-ovn-maintenance-worker-759f7d8784-7qqsx)

@nidzrai nidzrai changed the title feat: Confirm the OVN maintenance-worker periodic (ovn_maintenance_periodics) fires in our deployment feat: Confirm the OVN maintenance-worker periodic (ovn_maintenance_periodics) fires in our deployment PUC-2030 Sep 1, 2026
@nidzrai
nidzrai force-pushed the neutron_maintenance_worker_check branch 10 times, most recently from a185c2e to 67e2659 Compare September 4, 2026 10:26
@nidzrai
nidzrai force-pushed the neutron_maintenance_worker_check branch from 67e2659 to 9edf31f Compare September 4, 2026 10:41
@nidzrai
nidzrai added this pull request to the merge queue Sep 4, 2026
Merged via the queue into main with commit 9167102 Sep 4, 2026
69 checks passed
@nidzrai
nidzrai deleted the neutron_maintenance_worker_check branch September 4, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@haseebsyed12 haseebsyed12 haseebsyed12 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nidzrai @haseebsyed12